Cut Stems off the two types of chilies and remove as many seeds as you can. You can cut the larger Pasilla Chilies in halves or thirds. Boil in small pot of salted water for ten minutes or until soft. Drain saving the softened chilies and reserve the water. Take off any other exposed seeds that have appeared.
Place chilies, olive oil, key lime juice, garlic, honey, cilantro, brown sugar, and a bit of kosher salt and cracked black pepper in a food processor and pulse until well mixed and blended. If the mixture is too clumpy, add a bit of the reserved chili water.
Spread half of the mixture on all sides of the swordfish steaks to where the fish is well coated but not swimming in the glaze. Place in the refrigerator in a covered container for 1-3 hours to allow the flavors to intensify. Save the remaining portion of the glaze for basting while grilling.
Brush oil on grill grates to keep from sticking and turn heat to medium. Grill Sword steaks for about 8 minutes flipping halfway through or whenever grill marks appear. Baste while cooking with reserved marinade. Cooked medium the internal temperature should be approximately 135 degrees. Serve with your favorite summer vegetables or corn.